Add the space-discarding feature - #182

I've gone through the source code (except the emoji part). I believe the current algorithm is simple and robust enough. I only have some suggestions regarding doc comments and tests. See my individual comments.

More materials supporting the current algorithm

East Asian Width

UAX #14: Unicode Line Breaking Algorithm also uses East Asian Width to filter out East Asian characters.

The symbol $EastAsian stands for the set [\p{ea=F}\p{ea=W}\p{ea=H}] of characters with Fullwidth, Wide, or Halfwidth East Asian Width.

Pandoc

Pandoc's east_asian_line_breaks extension uses charWidth to determine if a soft break (e.g., single newline in markdown and typst) should be removed. However, we've argued in typst/typst#7350 (comment) that the rules for determining widths are too complicated and it's better to use East Asian Width directly.

LaTeX

LaTeX cannot be taken as a reference, because the implementations are limited by the technology. Specifically, the whitespace in 字\n“ should be discarded, but luatexja keeps it. And the whitespace in ”\nA should be kept as a word space, but xeCJK discards it. See typst/typst#792 (comment) for the tests.

Typst cjk-unbreak

As for typst packages, cjk-unbreak uses the following regex to determine if a character is CJ (Chinese + Japanese) and discards the space iff either side matches the regex.

[\p{Han}，。；：！？‘’“”（）「」【】…—\p{Hiragana}\p{Katakana}]

This package is designed only for CJ, so the regex includes a few characters that are considered YesOrAmbiguous in this PR.
The algorithm in this PR is designed for all writing systems, so the difference to cjk-unbreak is acceptable.

Typst cjk-spacer

A newer typst package, cjk-spacer, uses a more complex algorithm. If I understand correctly, then its algorithm is equivalent to the following.

#let default-cjk-regex = regex(
  "["
    + "\p{scx:Hira}\p{scx:Kana}\p{scx:Han}\p{scx:Hang}\p{scx:Bopo}"
    + "\u3000-\u303F" // CJK Symbols and Punctuation
    + "\u3190-\u319F" // Kanbun
    + "\u31C0-\u31EF" // CJK Strokes
    + "\u3200-\u32FF" // Enclosed CJK Letters and Months
    + "\u3300-\u33FF" // CJK Compatibility
    + "\uFE10-\uFE1F" // Vertical Forms
    + "\uFE30-\uFE4F" // CJK Compatibility Forms
    + "\uFE50-\uFE6F" // Small Form Variants
    + "\uFF00-\uFFEF" // Halfwidth and Fullwidth Forms
    + "]",
)
#let default-western-open-punc-regex = regex(
  "[\p{Pi}\p{Ps}--["
    + "\u3000-\u303F" // CJK Symbols and Punctuation
    + "\uFE10-\uFE1F" // Vertical Forms
    + "\uFE30-\uFE4F" // CJK Compatibility Forms
    + "\uFE50-\uFE6F" // Small Form Variants
    + "\uFF00-\uFFEF" // Halfwidth and Fullwidth Forms
    + "]]",
)
#let default-western-close-punc-regex = regex(
  "[\p{Pf}\p{Pe}\p{Term}--["
    + "\u3000-\u303F" // CJK Symbols and Punctuation
    + "\uFE10-\uFE1F" // Vertical Forms
    + "\uFE30-\uFE4F" // CJK Compatibility Forms
    + "\uFE50-\uFE6F" // Small Form Variants
    + "\uFF00-\uFFEF" // Halfwidth and Fullwidth Forms
    + "]]",
)

#let discard_space_between(before, after) = {
  if after.matches(western-open-punc-regex).len() == 0 and after.starts-with(cjk-regex) {
    true
  } else if before.matches(western-close-punc-regex).len() == 0 and before.ends-with(cjk-regex) {
    true
  } else {
    false
  }
}

The cjk-spacer algorithm does not merely consider the characters immediately adjacent to the space, but rather the text segments around the space. This approach is appropriate when typesetting a document, but it's too surprising at the syntax level.

Also, cjk-spacer treats K the same as CJ. According to previous feedbacks in typst/typst#7350, discarding spaces is not preferable for Korean texts.

And I haven't check if the Unicode blocks enumerated by cjk-spacer are equivalent to this PR, but I think it's worth checking before merging this PR.

Logic-wise, this seems reasonable and very well supported in terms of rationale. I'd trust in the research and the other involved people's judgement. I just have some minor nits.